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Abstract 


Writer: “Agent violates theme.” 


Through a particular choice of a predicate 
(e.g., “x violated y”), a writer can subtly 
connote a range of implied sentiments and 
presupposed facts about the entities x and 
y: (1) writer's perspective : projecting x 
as an “antagonist” and y as a “victim”, (2) 
entities' perspective: y probably dislikes 
x , (3) effect : something bad happened to y, 
(4) value: y is something valuable, and (5) 
mental state: y is distressed by the event. 

We introduce connotation frames as a rep¬ 
resentation formalism to organize these 
rich dimensions of connotation using typed 
relations. First, we investigate the fea¬ 
sibility of obtaining connotative labels 
through crowdsourcing experiments. We 
then present models for predicting the con¬ 
notation frames of verb predicates based 
on their distributional word representations 
and the interplay between different types 
of connotative relations. Empirical results 
confirm that connotation frames can be in¬ 
duced from various data sources that reflect 
how people use language and give rise to 
the connotative meanings. We conclude 
with analytical results that show the poten¬ 
tial use of connotation frames for analyzing 
subtle biases in online news media. 


1 Introduction 


People commonly express their opinions through 
subtle and nuanced language (Thomas et al., 2006[ 
Somasundaran and Wiebe^JZOlO). Often, through 


seemingly objective statements, the writer can in¬ 
fluence the readers’ judgments toward an event and 
their participants. Even by choosing a particular 
predicate, the writer can indicate rich connotative 
information about the entities that interact through 



8 (agent) 


8 (theme) 


Effect: the agent is not really 
affected by the violation 


Effect: the theme 
has been hurt 


Figure 1: An example connotation frame of “violate” as a 
set of typed relations: perspective V(x y), effect 8(x), 
value V(tr), and mental state S(x). 

the predicate. Specifically, through a simple state¬ 
ment such as “x violated y”, the writer can convey: 

(1) writer’s perspective: the writer is projecting 

x as an “antagonist” and y as a “victim”, elic¬ 
iting negative perspective from readers toward 
x (i.e., blaming x) and positive perspective 
toward y (supportive or sympathetic to y). 

(2) entities’ perspective: y most likely feels neg¬ 

atively toward x as a result of being violated. 

(3) effect: something bad happened to y. 

(4) value: y is something valuable, since it does 

not make sense to violate something worthless. 
In other words, the writer is presupposing y’s 
positive value as a fact. 

(5) mental state: y is most likely unhappy about 

the outcome Q 

lr To be more precise, y is most likely in a negative mental 
state assuming it is an entity that can have a mental state. 






















Verb 

Subset of Typed Relations 

Example Sentences 

L/R 

suffer 

V(w -A agent) = + 

V(w —»• theme) = — 

V{ agent —>• theme) = — 

£ (agent) = - 
V(agent) = + 

<5 (agent) = - 

The story begins in Illinois in 1987, when a 17- 
year-old girl suffered a botched abortion. 

R 

guard 

V(w —»• agent) = + 

V(w -A theme) = + 

V( agent theme) — + 

8 (theme) — + 

V (theme) — + 

S (theme) — + 

In August, marshals guarded 25 clinics in 18 
cities. 

L 

uphold 

V(w -A theme) = + 

V( agent theme) — + 

8 (theme) — + 

V (theme) — + 

A hearing is scheduled to make a decision on 
whether to uphold the clinic’s suspension. 

R 


Table 1: Example typed relations (perspective V(x y ), effect £(x), value V(x), and mental state <S(x)), where w denotes 
the writer. Not all typed relations are shown due to space constraints. The example sentences demonstrate the usage of the 
predicates in left [L] or right [R] leaning news sources. 


Even though the writer might not explicitly state 
any of the interpretations [1-5] above, the readers 
will be able interpret these intentions as a part of 
their comprehension. In this paper, we present an 
empirical study of how to represent and induce the 
connotative interpretations that can be drawn from 
a verb predicate, as illustrated above. 

We introduce connotation frames as a represen¬ 
tation framework to organize the rich dimensions 
of the implied sentiment and presupposed facts. 
Figure [I] shows an example of a connotation frame 
for the predicate violate. We define four different 
typed relations: V(x -A y) for perspective of x 
towards y, £{x) for effect on x, V(x) for value of 
x , and S(x) for mental state of x. These relation¬ 
ships can all be either positive (+), neutral (=), or 
negative (-). 

Our work is the first study to investigate frames 
as a representation formalism for connotative 
meanings. This contrasts with previous com¬ 
putational studies and resource development for 
frame semantics, where the primary focus was al¬ 
most exclusively on denotational meanings of lan¬ 
guage ( |Baker et al., 1998[jPalmer et al., 2005| ). Our 
formalism draws inspirations from the earlier work 
of frame semantics, however, in that we investi¬ 
gate the connection between a word and the related 


world knowledge associated with the word ([Fill 


more, 1976 ), which is essential for the readers to 


interpret many layers of the implied sentiment and 
presupposed value judgments. 

We also build upon the extensive amount of lit¬ 


erature in sentiment analysis (Pang and Fee, 2008 


Fiu and Zhang^O^), especially the recent emerg¬ 


ing efforts on implied sentiment analysis (Feng 


et al., 20T3| |Greene and Resnik, 2009| ), entity - 


entity sentiment inference ( [Wiebe and Deng, 2014| ), 
opinion role induction (Wiegand and Ruppenhofer, 
2015| ), and effect analysis ( [Choi and Wiebe, 2014] ). 


However, our work is the first to organize aspects of 
the connotative information into coherent frames. 

More concretely, our contributions are threefold: 
(1) a new formalism, model, and annotated dataset 
for studying the connotation frames from large- 
scale natural language data and statistics, (2) data- 
driven insights into the dynamics among different 
typed relations within each frame, and (3) an ana¬ 
lytic study to show the potential use of connotation 
frames for analyzing subtle biases in journalism. 

The rest of the paper is organized as follows: in 
§2, we provide the definitions and data-driven in¬ 
sights for connotation frames. In §3, we introduce 
models for inducing the connotation frames, fol¬ 
lowed by empirical results, annotation studies, and 
analysis on news media in §4. We discuss related 
work in §5 and conclude in §6. 

2 Connotation Frame 

Given a predicate v, we define a connotation frame 
F{y) as a collection of typed relations and their po¬ 
larity assignments: (i) perspective V v {x% -A Xj): 
whether the predicate v implies directed sentiment 
from the entity X{ to the entity xy, (ii) value V v {xf)\ 
whether X{ is presupposed to be valuable by the 
predicate v, (iii) effect £ v (xi): whether the event 
denoted by the predicate v is good or bad for the 
entity x*, and (iv) mental state S v {xi): the likely 
mental state of the entity X{ as a result of the event. 
We assume that each typed relation can have one of 
the three connotative polarities G {+, —, =}, i.e., 
positive, negative, or neutral. Our goal in this paper 
is to focus on the general connotation of the predi¬ 
cate considered out of context. We leave contextual 
interpretation of connotation as future work. 

Table [l] shows examples of connotation frame 
relations for the verbs suffer , guard , and uphold , 
along with example sentences. For instance, for the 
verb suffer , the writer is likely to have a positive 





































Verb 

x’s role 

V(yv —► •) 

Left-leaning Sources 

Right-leaning Sources 

accuse 

agent 

theme 

+ 

Putin, Progressives, Limbaugh, Gingrich 
official, rival, administration, leader 

activist, U.S., protestor, Chavez 
Romney, Iran, Gingrich, regime 

attack 

agent 

theme 

+ 

McCain, Trump, Limbaugh 
Gingrich, Obama, policy 

Obama, campaign, Biden, Israel 
citizen, Zimmerman 

criticize 

agent 

theme 

+ 

Ugandans, rival, Romney, Tyson 
Obama, Allen, Cameron, Congress 

Britain, passage, Obama, Maddow 
Pelosi, Romey, GOP, Republicans 


Table 2: Media Bias in Connotation Frames: Obama, for example, is portrayed as someone who attacks or criticizes others by 
the right-leaning sources, whereas the left-leaning sources portray Obama as the victim of harsh acts like “attack” and “criticize”. 


perspective towards the agent (e.g., being support¬ 
ive or sympathetic toward the “17-year-old girl” in 
the example shown on the right) and a negative 
perspective towards the theme (e.g., being negative 
towards ‘botched abortion”). 

2.1 Data-driven Motivation 

Since the meaning of language is ultimately contex¬ 
tual, the exact connotation will vary depending on 
the context of each utterance. Nonetheless, there 
still are common shifts or biases in the connota- 
tive polarities, as we found from two data-driven 
analyses. 

First, we looked at words from the Subjectivity 


Lexicon (Wilson et al., 2005) that are used in the 


argument positions of a small selection of predi¬ 
cates in Google Syntactic N-grams ( [Goldberg and| 
Orwant, 2013). For this analysis, we assumed that 
the agent is the word in the subject position while 
the theme is the word in the object position. We 
found 64% of the words in the agent role of suffer 
are positive, and 94% of the words in the theme 
role are negative, which is consistent with the po¬ 
larities of the writer’s perspective towards these 
arguments, as shown in Table [T] For guard , 57% 
of the agents and 76% of the themes are positive, 
and in the case of uphold , 56% of the agents and 
72% of the themes are positive. 

We also investigated how media bias can po¬ 
tentially be analyzed through connotation frames. 


From the Stream Corpus 2014 dataset ( |KBA, 2014] ), 
we selected all articles from news outlets with 
known political biases]^] and compared how they 
use polarised words such as “accuse”, “attack”, and 
“criticize” differently in light of V(w agent) 
and V(w -A theme) relations of the connota¬ 
tion frames. Table [2] shows interesting contrasts. 
Obama, for example, is frequently portrayed as 

2 The articles come from 30 news sources indicated by 
others as exhibiting liberal or conservative leanings ( [Mitchell 


et al., 2014][Center for Media and Democracy, 2013[|Center 

for Media and Democracy, 2012 HWC Library, 201 lp 


someone who attacks or criticizes others accord¬ 
ing to the right-leaning sources, whereas the left¬ 
leaning sources portray Obama as the victim of 
harsh acts like “attack” or “criticize” 0 Further¬ 
more, by knowing the perspective relationships 
V(w -A Xi) associated with a predicate, we can 
make predictions about how the left-leaning and 
right-leaning sources feel about specific people or 
issues. For example, because left-leaning sources 
frequently use McCain, Trump, and Limbaugh in 
the agent position of attack, we might predict that 
these sources have a negative sentiment towards 
these entities. 


2.2 Dynamics between Typed Relations 


Given a predicate, the polarity assignments of typed 
relations are interdependent. For example, if the 
writer feels positively towards the agent but nega¬ 
tively towards the theme, then it is likely that the 
agent and the theme do not feel positively towards 
each other. This insight is related to that of |Wiebe 


and Deng (2014| ), but differs in that the polarities 


are predicate-specific and do not rely on knowledge 
of prior sentiment towards the arguments, them¬ 
selves. This and other possible interdependencies 
are summarized in Table [3] These interdependen¬ 
cies serve as general guidelines of what properties 
we expect to depend on one another, especially in 
the case where the polarities are non-neutral. We 
will promote these internal consistencies in our 
factor graph model (Q as soft constraints. 

There also exist other interdependencies that we 
will use to simplify our task. First, the directed 
sentiments between the agent and the theme are 
likely to be reciprocal, or at least do not directly 
conflict with + and — simultaneously. This intu¬ 
ition follows from a notion of balance derived by 
social theory ( [Heider, 1946| . Therefore, we assume 


3 That is, even if someone truly deserves criticism from 
Obama, left-learning sources would choose slightly different 
wordings to avoid a potentially harsh portrayal of Obama. 



































Perspective Triad: If argument x% is positive towards Xj, and Xj is positive towards Xk, then we expect xi is also 
positive towards Xk. Similar dynamics hold for the negative case. 

Pw — yx ^ ~ 1 (Pw —j 0 Pxi^-Xj) 

Perspective - Effect: If a predicate has a positive effect on one argument, then we expect that the interaction 
between the arguments was positive. Similar dynamics hold for the negative case. 

- Pxj—>Xi 

Perspective - Value: If argument Xi is presupposed as valuable, then we expect that the writer also views Xi 
positively. Similar dynamics hold for the negative case. 

- P W^-Xl 

Effect - Mental State: If the predicate has a positive effect on an argument Xi, then we expect that Xi will gain a 
positive mental state. Similar dynamics hold for the negative case. 

S Xi £ Xi 


Table 3: Potential Dynamics among Typed Relations: we propose models that parameterize these dynamics using log-linear 
models (frame-level model in §3). 



Figure 2: A factor graph for predicting the polarities of the 
typed relations that define a connotation frame for a given 
verb predicate. The factor graph also includes unary factors 
(ipemb ), which we left out for brevity. 


that V(xi -0 Xj) = V(xj -0 xi) — V(xi <(0 xj), 

and we only measure for these binary relationships 
going in one direction. In addition, we assume the 
predicted^ perspective from the reader r to an ar¬ 
gument, V(r -0 x), is likely to be the same as the 
implied perspective from the writer w to the same 
argument, V(w -0 x). So, we only try to learn the 
perspective of the writer. Lifting these assumptions 
will be future work. 

For simplicity, our work only explores verb pred¬ 
icates and focuses on the polarities involving the 
agent and the theme roles, which we will refer to 
as a and t. We will assume that these roles are 
correlated to the subject and object positions. 


4 Surely different readers can and will form varying opin¬ 
ions after reading the same text. Here we concern with the 
most likely perspective of the general audience, as a result of 
reading the text. 


3 Modeling Connotation Frames 


Our task is essentially that of lexicon induc- 
tion ( [Akkaya et al., 2009) [Feng et al., 2013| ) in that 
we want to induce the connotation frames of previ¬ 
ously unseen verbs. For each verb predicate, we in¬ 
fer a connotation frame composed of 9 relationship 
aspects that represent: perspective {V(w -0 t), 
V(w -0 a), V(a -0 t)}, effect {£(£), 8(a)}, value 
{V(t), V(a)}, and mental state {S(t), S(a)} po¬ 
larities, where w, a, t denote the writer, the agent, 
and the theme, respectively. 

We propose two models: an aspect-level model 
that makes the prediction for each typed relation 
independently based on the distributional represen¬ 
tation of the context in which the predicate com¬ 
monly appears ( §3.1| ), and a frame-level model that 
makes the prediction over the connotation frame 
collectively in consideration of the dynamics be¬ 


tween typed relations (£3.2). 


3.1 Aspect-Level 

Our aspect-level model predicts labels for each of 
these typed relations separately. As input, we use 
the 300-dimensional dependency-based word em¬ 
beddings from |Levy and Goldberg (2014} ). For each 
aspect, there is a separate MaxEnt (maximum en¬ 
tropy) classifier used to predict the label of that as¬ 
pect on a given word-embedding, which is treated 
as a 300 dimensional input vector to the classi¬ 
fier. The MaxEnt classifiers learn their weights 
using LBFGS on the training data examples with 
re-weighting of samples to maximize for the best 
average FI score. 


3.2 Frame-Level 

Next we present a factor graph model (Figure [2]) 
of the connotation frames that parameterizes the 
dynamics between typed relations. Specifically, 




















for each verb predicate, the factor graph contains 
9 nodes representing the different aspects of the 
connotation frame involving the writer ( w ), the 
agent (a), and the theme (£). All these variables 
take polarity values from the set {—, =, +}. 

We define Y^ := {T* w t, T * wa , 'Pat: £ti SaiVti 
St, S a } as the set of relational aspects for the i th 
verb predicate. The factor graph for Y^, is illus¬ 
trated in Figure [2| and we describe the factor po¬ 
tentials, ip, in detail in the rest of this section. The 
probability of an assignment of polarities to the 
nodes in Y^ is: 

P(Y^) OC 'l/jpvi'Pwa, Va) 'ifav(Vwt, Vt) 

IpFE{Vat, So) ^PE {Vat, St) 

^ES (£cn<Sa) lhs{£tiSt) 

, 0Pt(P wti 'Pwai 'Pat) | emb{y ) 

yeY z 

Embedding Factors We include unary factors on 
all nodes to represent the results of the aspect-level 
classifier. Incorporating this knowledge as factors, 
as opposed to fixing the variables as observed, af¬ 
fords us the flexibility of representing noise in the 
labels as soft evidence. The potential function ip em b 
is a log-linear function of a feature vector f, which 
is a one-hot feature vector representing the polarity 
of a node (+,—,or =). For example, with the node 
representing the value of the theme (Vt): 

TpembiVt) = e 0 v t' /(Vf) 

The potential ip em b is defined similarly for the re¬ 
maining eight nodes. 

Weights 6 are learned in a piecewise likelihood 
manner ( [Sutton and McCallum, 2009) for each fac¬ 
tor independently using stochastic gradient descent 
(SGD) over the training data. 

Interdependency Factors We include interde¬ 
pendency factors to promote the properties defined 
by the dynamics between relations ( §2.2| ). The po¬ 
tentials for Perspective Triad, Perspective-Value, 
Perspective-Effect, and Effect-State Relationships 
pipPT, 'ippv, P^pe, r pES respectively) are all defined 
using log-linear functions of one-hot feature vec¬ 
tors that encode the combination of polarities of 
the neighboring nodes. Thus the potential ippp is: 

PT(Vwt,Vwa,V at ) = 

And we define the potentials for ippy, ippE, and r pES 
for nodes pertaining to the agent as: 

^pv{V wa ,Va) = e*PV,«-/(7W.) 


ipPE(V at ,S a ) = e 9pE *-H' p M 

il>Es{Sa,S a ) = e e *s*W«£«) 

and we define the potentials for the theme nodes 
similarly. As with the unary seed factors, weights 
6 are learned using SGD over training data. 

Belief Propagation We use belief propagation 
to induce the connotation frames of previously un¬ 
seen verbs. In the belief propagation algorithm, 
messages are iteratively passed between the nodes 
to their neighboring factors. Each message /i, con¬ 
taining a scalar for each value x G {—,=,+}, is 
defined from each node v to a neighboring factor / 
as follows: 

Vv-^f 0) oc P[ 

f*EN(v)\a 

and from each factor a to a neighboring node v as: 

M/-m> oc ip(a/) IJ /V^/04*) 

x',x' v =x v*EN(f)\v 

Our factor graph does not contain any loops, so we 
are able to perform exact inference by choosing a 
root node and performing message passing from 
the leaves to the root and back to the leaves. At 
the conclusion of message passing, the probability 
of a specific polarity associated with node v being 
equal to x is proportional to ri/ejv^) 

4 Experiments 

We first describe crowd-sourced annotations ( §4.1| ), 
then present the empirical results of predicting con¬ 
notation frames ( §4.2| ), and conclude with qualita¬ 
tive analysis of a large corpus ( §4.3| ). 

4.1 Data and Crowdsourcing 

In order to understand how humans interpret conno¬ 
tation frames, we designed an Amazon Mechanical 
Turk (AMT) annotation study. We gathered a set of 
transitive verbs commonly used in the New York 
Times corpus (Sandhaus, 2008), selecting the 2400 
verbs that are used more than 200 times in the cor¬ 
pus. Of these, AMT workers annotated the 1000 
most frequently used verbs. 

Annotation Design In a pilot annotation exper¬ 
iment, we found that annotators have difficulty 
thinking about subtle connotative polarities when 
shown predicates without any context. Therefore, 
we designed the AMT task to provide a generic 
context as follows. We first split each verb predi¬ 
cate into 5 separate tasks that each gave workers a 






different generic sentence using the verb. To cre¬ 
ate generic sentences, we used Google Syntactic 
N-grams ( [Goldberg and Orwant, 2013| ) to come 
up with a frequently seen Subject-Verb-Object tu¬ 
ple which served as a simple three-word sentence 
with generic arguments 0 For each of the 5 sen¬ 
tences, we asked 3 annotators to answer questions 
like “How do you think the agent feels about the 
event described in this sentence?” In total, each 
verb has 15 annotations aggregated over 5 different 
generic sentences containing the verb. 

In order to help the annotators, some of the ques¬ 
tions also allowed annotators to choose sentiment 
using additional classes for “positive or neutral” 
or “negative or neutral” for when they were less 
confident but still felt like a sentiment might ex¬ 
ist. When taking inter-annotator agreement, we 
count “positive or neutral” as agreeing with either 
“positive” or “neutral” classes. 

Annotator agreement Table [4] shows agreements 
and data statistics. The non-conflicting (NC) agree¬ 
ment only counts opposite polarities as disagree¬ 
ment^] From this study, we can see that non-expert 
annotators are able to see these sort of relationships 
based on their understanding of how language is 
used. From the NC agreement, we see that annota¬ 
tors do not frequently choose completely opposite 
polarities, indicating that even when they disagree, 
their disagreements are based on the degree of con¬ 
notations rather than the polarity itself. The average 
Krippendorff alpha for all of the questions posed 
to the workers is 0.25, indicating stronger than ran¬ 
dom agreement. Considering the subtlety of the 
implicit sentiments that we are asking them to anno¬ 
tate, it is reasonable that some annotators will pick 
up on more nuances than others. Overall, the per¬ 
cent agreement is encouraging that the connotative 
relationships are visible to human annotators. 
Aggregating Annotations We aggregated over 
crowdsourced labels (fifteen annotations per verb) 
to create a polarity label for each aspect of a verbj^] 

Final distributions of the aggregated labels are 
included in the right-hand columns of Table]?] No- 


5 Because Google Syntactic N-grams only provide depen¬ 
dency types and do not provide semantic roles, we approxi¬ 
mate the agent and theme as the subject and the object respec¬ 
tively. 

6 Annotators were asked yes/no questions related to Value, 
so this does not have a corresponding NC agreement score. 

7 We take the average to obtain scalar value be¬ 
tween [—1., 1.] for each aspect of a verb’s connotation frame. 
For simplicity, we cutoff the ranges of negative, neutral 
and positive polarities as [-1,-0.25), [—0.25,0.25] and 
(0.25,1], respectively. 


Aspect 

% Agreement 

Distribution 

Strict 

NC 

% + 

%- 

V(w t) 

75.6 

95.6 

36.6 

4.6 

V(w —»• a) 

76.1 

95.5 

47.1 

7.9 

Via t) 

70.4 

91.9 

45.8 

5.0 

m 

52.3 

94.6 

50.3 

20.24 

£{a) 

53.5 

96.5 

45.1 

4.7 

m 

65.2 

- 

78.64 

2.7 

V(o) 

71.9 

- 

90.32 

1.4 

S(t) 

79.9 

98.0 

12.8 

14.5 

S(a) 

70.4 

92.5 

50.72 

8.6 


Table 4: Label Statistics: % Agreement refers to pairwise 
inter-annotator agreement. The strict agreement counts agree¬ 
ment over 3 classes (“positive or neutral” was counted as 
agreeing with either + or neutral), while non-conflicting (NC) 
agreement also allows agreements between neutral and -/+ (no 
direct conflicts). Distribution shows the final class distribution 
of -/+ labels created by averaging annotations. 


tably, the distributions are skewed toward positive 
and neutral labels. The most skewed connotation 
frame aspect is the value V{x) which tends to be 
positive, especially for the agent argument. This 
makes some intuitive sense since, as the agent ac¬ 
tively causes the predicate event to occur, they most 
likely have some intrinsic potential to be valuable. 
An example of a verb where the agent was labelled 
with negative value is “contaminate”. In the most 
generic case, the writer is using “contaminate” to 
frame the agent as being worthless (and even harm¬ 
ful) with regards to the other event participants. For 
example, in the sentence “his touch contaminated 
the food,” it is clear that the writer presupposes “his 
touch” to be of negative value in how it impacts the 
rest of the event. 

4.2 Connotation Frame Prediction 

Using the crowdsourced labels, we randomly di¬ 
vide the annotated verbs into training, dev, and 
held-out test sets of equal size (300 verbs each). 
For evaluation we measure average accuracy and 
FI score of induced labels for the 9 different conno¬ 
tation frame relationship types for which we have 
annotations: V(w -A £), V(w —^ a), V(a -A £), 
V(£), V(a), £(£), £(a), <S(£), and S(a), where w 
refers to the writer, a to the agent, and t to the 
theme. 

Baselines To show the non-trivial challenge of 
learning Connotation Frames, we include a simple 
majority-class baselines. The Majority classi¬ 
fier assigns each of the 9 relationships the label 
of the majority of that relationship type found in 
the training data. Some of these relationships (in 









particular, the Value of agent/theme) have skewed 
distributions, so we expect this classifier to achieve 
a much higher accuracy than random but a much 
lower overall FI score. 


Additionally, we add a Graph Prop baseline 
that is comparable to algorithms like graph prop¬ 
agation or label propagation which are often used 
for (sentiment) lexicon induction ( [Velikovich et al.J 


2010a). We use a factor graph with nodes repre¬ 


senting the polarity of each typed relation for each 
verb. Binary factors connect nodes representing 
a particular type of relation for two similar verbs 
(e.g. V(w -A t) for verbs persuade and convince). 
These binary factors have hand-tuned potentials 
that are proportional to the cosine similarity of the 
verbs’ embeddings, encouraging similar verbs to 
have the same polarity for the various relational 
aspects. We use words in the training data as the 
seed set and use loopy belief propagation to propa¬ 
gate polarities from known nodes to the unknown 
relationships. 


Finally, we use a 3-Nearest Neighbor base¬ 
line that labels relationships for a verb based on 
the predicate’s 300-dimensional word embedding 
representation, using the same embeddings as in 
our aspect-level. 3-Nearest Neighbor labels 
each verb using the polarities of the three closest 
verbs found in the training set. The most similar 
verbs are determined using the cosine similarity 
between word embeddings. 

Results As shown in Table [5j aspect-level and 
frame-level models consistently outperform all 
three baselines — Majority, 3-NN, Graph 
Prop in the development set across the different 
types of relationships. In particular, the improved 
FI scores show that these models are able to per¬ 
form better across all three classes of labels even 
in the most skewed cases. The frame-level model 
also frequently improves the FI scores of the la¬ 
bels from what they were in the aspect-level model. 
The summarized comparison of the classifiers’ per¬ 
formance test set is shown in Table |6j As with 
the development set, aspect-level and frame-level 
are both able to outperform the baselines. Fur¬ 
thermore, the frame-level formulation is able to 
make improvement over the results of the aspect- 
level classification, indicating that the modelling of 
inter-dependencies between relationships did help 
correct some of the mistakes made. 


One point of interest about the frame-level re¬ 
sults is whether the learned weights over the consis- 
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Figure 3: Learned weights, 0, of embedding factor for 
the perspective of agent to theme J3§ and the weights the 
perspective triad (PT) factor Pb|3d> . Lighter shades are for 
weights that are more positive, whereas dark blue is more 
negative. 


tency factors match our initial intuitions about inter¬ 
dependencies between relationships. The weights 
learned in our algorithm do tell us something in¬ 
teresting about the degree to which these inter¬ 
dependencies are actually found in our data. 

We show the heat maps for some of the learned 
weights in Figure [3] In [3a] we show the weights of 
one of the embedding factors, and how the node’s 
polarities are more strongly weighted when they 
match the aspect-level output. In the rest of the 
figure, we show the weights for the other per¬ 
spective relationships when 7 D (w t) is nega¬ 

tive ( [3b] ), neutral ( [3c] ), and positive ( [3d] ), respec¬ 
tively. Based on the expected interdependencies, 
when V(w -A t) : —, the model should favor 
V(w -A a) / V(a -A t) and when V(w A t) : +, 
the model should favor V(w -A a) = V(a -A t). 
Our model does, in fact, learn a similar trend, with 
slightly higher weights along these two diagonals in 
the maps 3b and 3d Interestingly, when V(w -A t) 
is neutral, weights slightly prefer for the other two 
perspectives to resemble one another, but with high¬ 
est weights being when other perspectives are also 
neutral. 


4.3 Analysis of a Large News Corpus 

Using the connotation frame, we present measured 
implied sentiment in online journalism. 

Data From the Stream Corpus ( jKBA, 20l4] ), 
we select 70 million news articles. We extract 
subject-verb-object relations for this subset us- 































Aspect 

Algorithm 

Acc. 

AvgFi 


Majority 

56.52 

24.07 


Graph Prop 

59.53 

50.20 

V(w —y t ) 

3-nn 

62.88 

47.93 


Aspect-Level 

67.56 

56.18 


Frame-Level 

67.56 

56.18 


Majority 

49.83 

22.17 


Graph Prop 

52.84 

42.93 

V(w —y a) 

3-nn 

55.18 

45.88 


Aspect-Level 

60.54 

60.72 


Frame-Level 

61.87 

63.07 


Majority 

49.83 

22.17 


Graph Prop 

52.17 

46.57 

V(a —y t) 

3-nn 

56.52 

52.94 


Aspect-Level 

63.21 

61.70 


Frame-Level 

63.88 

62.56 


Majority 

48.83 

21.87 


Graph Prop 

54.85 

51.40 

S(t) 

3-nn 

55.18 

51.53 

Aspect-Level 

64.21 

63.63 


Frame-Level 

65.22 

64.67 


Majority 

49.83 

22.17 


Graph Prop 

52.17 

35.56 

£(a) 

3-nn 

54.85 

42.63 


Aspect-Level 

62.54 

53.82 


Frame-Level 

63.88 

56.81 


Majority 

79.60 

29.55 


Graph Prop 

71.91 

35.10 

m 

3-nn 

76.25 

39.09 


Aspect-Level 

75.92 

45.45 


Frame-Level 

76.25 

48.13 


Majority 

89.30 

31.45 


Graph Prop 

84.62 

38.82 

V{a) 

3-nn 

85.62 

38.45 

Aspect-Level 

87.96 

48.06 


Frame-Level 

87.96 

48.06 


Majority 

71.91 

27.89 


Graph Prop 

69.90 

55.57 

sit) 

3-nn 

72.91 

59.26 


Aspect-Level 

81.61 

72.85 


Frame-Level 

81.61 

72.85 


Majority 

50.84 

22.47 


Graph Prop 

48.83 

35.40 

S)a) 

3-nn 

54.85 

45.51 


Aspect-Level 

61.54 

53.88 


Frame-Level 

61.54 

53.88 


Table 5: Detailed breakdown of results on the de¬ 
velopment set using accuracy and average FI over 
the three class labels (+,-,=). 


Algorithm 

Acc. 

Avg Fi 

Graph Prop 

58.81 

41.46 

3-nn 

63.71 

47.30 

Aspect-Level 

67.93 

53.17 

Frame-Level 

68.26 

53.50 



Democrat 


Figure 4: Average sentiment of Democrats and Republicans 
(as agents) to selected nouns (as their themes), aggr egated 
over a large corpus using the learned lexicon ( §4.2| ». The 
line indicates identical sentiments, i.e. Republicans are more 
positive towards the nouns that are above the line. 

ing the direct dependencies between noun phrases 
and verbs as identified by the BBN Serif system, 
obtaining 1.2 billion unique tuples of the form 
(url,subject,verb,object,count). We also extract tu¬ 
ples from news articles from the Annotated English 
Gigaword Corpus (Napoles et al., 2 012| ), which 
contains nearly 10 million articles, resulting in an 
additional 120 million unique tuples. 

Estimating Entity Polarities Using connotation 
frames, we can also measure entity-to-entity sen¬ 
timent at a large scale. Figure |4} for example, 
presents the polarity of entities “Democrats” and 
“Republicans” towards a selected set of nouns, by 
computing the average estimated V(a -A t) polar¬ 
ity (using our frame-level output) over triples where 
one of these entities appears as part of the phrase 
in the agent role (e.g. “Democrats” or “Republican 
party”). Apart from nouns that both entities are pos¬ 
itive (“business”, “constitution”) or negative (“the 
allegations”,“veto threat”) towards, we can also 
see interesting examples in which Democrats feel 
more positively (below the line: “nancy pelosi”, 
“unions”, “gun control”, etc.) and ones where Re¬ 
publicans feel more positive (“the pipeline”, “gop 
leadership”, “budget cuts”, etc.). Also, both enti¬ 
ties are neutral towards “idea” and “the proposal”, 
which probably owes to the fact that ideas or pro¬ 
posals can be good or bad for either entity depend¬ 
ing on the context. 


Table 6: Performance on the test set. Results are 
averaged over the different aspects. 




















5 Related Work 


Most prior work on sentiment lexicons focused on 
the overall polarity of words without taking into 


2005; Baccianella et al., 2010; Wiebe et al., 2005; 

Velikovich et al., 2010b 

Kaji and Kitsuregawa, 

2007; Kamps et al., 2004 

; Takamura et al., 2005; 


Adreevskaia and Bergler, 2006] ). Several recent 
studies began exploring more specific and nuanced 


aspects of sentiment such as connotation (Feng et 


al., 2013| ), good and bad effects (Choi and Wiebe, 


2014), and evoked sentiment (Mohammad and Tur¬ 


ney, 2010). Drawing inspirations from them, we 


present connotation frames as a unifying represen¬ 
tation framework to encode the rich dimensions of 
implied sentiment, presupposed value judgements, 
and effect evaluation, and propose a factor graph 
formulation that captures the interplay among dif¬ 
ferent types of connotation relations. 

Goyal et al. ( |2010a| |2010b[ ) investigated how 
characters (protagonists, villains, victims) in chil¬ 
dren’s stories are affected by certain predicates, 
which is related to the effect relations studied in this 
work. While Klenner et al. (2014) similarly investi¬ 
gated the relation between the polarity of the verbs 
and arguments, our work introduces new perspec¬ 
tive types and proposes a unified representation and 


inference model. Wiegand and Ruppenhofer (2015) 


also looked at perspective-based relationships in¬ 
duced by verb predicates with a focus on opinion 
roles. Building on this concept, our framework 
also incorporates information about the perspec¬ 
tives’ polarities as well as information about other 
typed relations. There have been growing interests 
for modeling framing ( Greene and Resnik, 2009; 


Hasan and Ng, 2013] ), biased language (Recasens 


et al., 2013] ) and ideology detection ( |Yano et aL\ 


2010). All these tasks are relatively less studied, 


and we hope our connotation frame lexicon will be 
useful for them. 

Sentiment inference rules have been explored 
by the recent work of Wiebe and Deng (2014 ) 
and [Deng and Wiebe (2014| . The focus of their 
work was on general inference rules that are not 
predicate-specific. In contrast, our work focuses 
on the notion that connotative polarities can be de¬ 
termined directly from the predicate, rather than 
partial knowledge of the arguments or the context 
in which it is being used. In brief, we make a 
novel conceptual connection between inferred sen¬ 
timents and frame semantics, organized as conno¬ 


tation frames, and present a unified model that inte¬ 
grates different aspects of the connotation frames. 

Finally, in a broader sense, what we study as 
connotation frames draws a connection to schema 
and script theory ( [Schank and Abelson, 1975[ ). Un¬ 
like prior work that focused on directly observable 
actions ( [Chambers and Jurafsky, 2009| Frermann 


|et al., 20~T4t [Bethard et al., 2008] ), we focus on 
implied sentiments that are framed by predicate 
verbs. 


6 Conclusion 

In this paper, we presented a novel system of 
connotative frames that define a set of implied 
sentiment and presupposed facts for a predicate. 
Our work also empirically explores different meth¬ 
ods of inducing and modelling these connotation 
frames, incorporating the interplay between rela¬ 
tions within frames. Our work suggests new re¬ 
search avenues on learning connotation frames, 
and their applications to deeper understanding of 
social and political discourse. All the learned 
connotation frames and annotations are available 
at http://homes.cs.Washington.edu/ 
-hrashkin/connframe.html, 
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